The Formation and Evolution of Faultlines in Large-Scale, Multi-Party Information Systems Development

Information systems development (ISD) projects are becoming more complex, larger in scale and scope, more distributed, and span a more diverse user base crossing multiple organizational boundaries. Findings from prior ISD research may not generalize well to this setting. Our research-in-progress addresses this gap by examining the participation of diverse stakeholders in a large-scale, multi-party governmentacademic-industrial ISD project over a four-year timeframe. We apply the theoretical lens of faultlines to understand how the participants work together during the project. Faultlines are hypothetical attributes (such as gender or expertise) that divide groups, and can cause conflict. Using network analysis we identify faultlines between the participants in the ISD project, and examine the faultlines over the course of the project. Our findings suggest the strong presence of faultlines early in the project; however, at least for the core group of participants, the faultlines dissipate over time as the group becomes more inclusive. Approaches to Information Systems Development Track Twenty-eighth International Conference on Information Systems Montreal 2007 1 The Formation and Evolution of Faultlines in LargeScale, Multi-Party Information Systems Development ** Research-in-Progress ** Introduction Information Systems Development (ISD) projects are becoming increasingly more complex in scale and scope, more distributed, and involve a more diverse user base (Levina 2005). Findings from past research in ISD and in user involvement have often focused on systems with a smaller and less diverse user group, usually located within a single organization (e.g., McKeen et al. 1994; Sabherwal and Robey 1993), and do not necessarily generalize to large scale projects that also involve a diverse user base of individual and institutional stakeholders. Studies on open source software (OSS) have examined the distributed aspect of systems development; however, developers, users and stakeholders of systems developed outside the OSS movement do not have a common set of values (e.g. Stewart and Gosain 2006) that create commitment to a project through identification with a larger movement. Also, the involvement of multiple institutions, rather than just individual users and developers, has not been examined, even though investment in large-scale ISD has increased on the part of the government and through government-academic projects, such as the “e-Science” initiative in the United Kingdom and the Global Environment for Network Innovations (GENI) initiative funded by the National Science Foundation in the United States. Our research-in-progress attempts to address this gap by examining the participation of a diverse body of stakeholders in a large-scale, multi-party government-academic-industrial ISD project over a four-year timeframe. The ISD project had the goal of integrating information technology and systems with civil engineering to provide a distributed collection of laboratory facilities that could be used by engineers to perform model-based simulations of earthquakes. The theoretical lens applied in our study – faultlines – comes from the group diversity literature. Faultlines are hypothetical lines that divide groups, such as gender or expertise, and which may cause conflict among subgroups (Lau and Murnighan 1998). Using network analysis we identify faultlines between groups in the ISD project, and we examine how faultlines become salient or dissipate over time. The analysis reported here is a first step in an attempt to integrate the social network and social psychological perspectives in examining the process of large scale, multi-party information systems development. Theory and Research Questions A large body of research has attempted to examine the conditions under which group diversity has positive versus negative impacts on group-level and organizational outcomes, but results have been largely inconclusive (Williams and O’Reilly 1998). The faultlines model shifted the level of analysis so that diversity was examined as a group characteristic rather than just as an individual-level attribute (Lau and Murnighan 1998). Accordingly, the effects of diversity are hypothesized to vary with the group’s composition and structure. Faultlines, by definition, are “hypothetical dividing lines that may split a group into subgroups based on one or more attributes” (Lau and Murnighan 1998). Faultlines can be an important determinant of subgroup conflict, particularly when the group's task is relevant to one of its faultlines. Groups might have several attributes that distinguish their members and therefore several possible faultlines. For example, two such attributes are expertise and affiliation. Expertise and affiliation are valuediagnostic attributes (Tajfel and Turner 1986): individuals rely on those attributes in categorizing others into an ingroup and outgroup, and in deciding whether and how much to cooperate and share information with others. Also, expertise and affiliation are distinct and uncorrelated indices of group-level diversity and as such they allow us to examine the effect of diversity among the project stakeholders from multiple angles. Recent research in group diversity and performance has found that, by integrating multiple attributes, diversity examined from the faultlines perspective had greater explanatory power for group outcomes than single-attribute diversity (Lau and Murnighan 2005). This suggests the importance of investigating multiple faultlines, rather than focusing on a single one. 2 See the website http://www.rcuk.ac.uk/escience/default.htm for more information. 3 For information on GENI, see http://www.geni.net. Approaches to Information Systems Development Track Twenty-eighth International Conference on Information Systems Montreal 2007 2 Diversity of expertise is relevant to the context of our study because the coordination and execution of large ISD projects involves complex decisions and tasks that in turn involve the input of individuals with different backgrounds (or knowledge domains, such as geology, information technology, systems design, usability) that represent distinct “thought worlds” (Dougherty 1992). There are many instances in the IS literature of expertise diversity causing conflict during ISD, for example, users and developers battling over design and implementation issues (Urquhart 2001; Barki and Hartwick 1994; Newman and Noble 1990; Markus 1983). Thus it is conceivable that an expertise faultline may develop in large-scale, multi-party ISD efforts. Affiliation is also relevant in the context of large-scale multi-party ISD projects, in which stakeholders are likely to be affiliated with different organizational units or firms. Prior studies that have looked at the diversity of affiliations within a single organization (business unit affiliations) have shown that different affiliations within a team operate as category labels that members rely on in the course of their work with others so that assumptions about values and goals are transferred from the unit to the individual representing that unit (Bunderson and Sutcliffe 2000). Members’ affiliations trigger social categorization processes (Early and Mosakowski 2000) so that members tend to favor, cooperate and align with those they perceive to have similar (or the same) affiliation as them. It is easy to imagine how this might be salient in the context of ISD projects, where a faultline might develop between users from business departments and developers from IS departments (cf Kirsch and Haney 2006; Curtis et al 1988). Moreover, when ISD projects span organizations (such as government, academic, and industrial organizations, as in our research site), it is plausible that a faultline may occur along the affiliation boundary, with academics forming one group and government officials forming another. Though it seems that faultlines may erupt due to expertise and affiliation diversity, there is little research in this area from which to develop a specific research model. Moreover, it seems plausible that faultlines may also stem from other attributes that divide groups, such as gender, age, or ethnicity. In addition, as a project unfolds over time, faultlines may change, for example, differences in expertise or affiliation may become less salient as individuals work closely with each other. Thus, to explore the involvement of different stakeholder groups in the development of a large-scale, multi-party effort, we posed two broad research questions: What faultlines form between groups of stakeholders involved in an ISD project? How do the faultlines change as the project evolves? Research Site and Context The source of our data comes from the George E. Brown, Jr. Network for Earthquake Engineering Simulation (NEES), a project created by the National Science Foundation (NSF) to better understand earthquakes, their causes and effects. The goal of NEES is to promote collaborative research, shared understanding, and shared equipment (including engineering equipment sites, computational resources, and digital libraries) among the earthquake engineering community. The NEES project involved the creation of NEESgrid, a large-scale network designed to connect researchers and experimental equipment sites comprised of advanced earthquake testing capabilities. The NEESgrid infrastructure and systems development project involves tasks that require inputs from multiple knowledge domains, and involve multiple user groups with different needs and requirements. The user groups and governing committees associated with NEES (including NEESgrid) are comprised of academics and professionals with different expertise backgrounds and different institutional affiliations (academia, private sector, and NSF). Given that NEES includes a community of developers, users and institutional stakeholders, the faultlines approach is an appropriate group-level perspective as it allows us to examine the structure of participation of multiple groups within the NEES community and the effect of the groups’ diversity on the NEES project as it unfolded over time. Data Sources and Analysis The data were obtained by examining official board meeting and committee meeting notes obtained from the NEES Governance web site from 2003 to 2006. The notes were examined and individuals were included as participants if they either participated in an actual meeting or were mentioned in the meeting notes as having involvement in the NEES project. The data obtained consisted of 149 individuals who had attended meetings in 2003, 2004, 2005, and 2006. Because the NEES Board of Directors had only one meeting in 2004 and there were no committee meetings in 2004, the 2003 and 2004 data were combined into a single data set. The data showed that 4 For more information, see www.nees.org. Approaches to Information Systems Development Track Twenty-eighth International Conference on Information Systems Montreal 2007 3 149 individuals collectively attended a total of 34 different meetings over the four year period, with 658 individual attendees/mentions during that time period. During the 2003 to 2006 time frame, several committees were active in the NEES project. Overseeing the project is the NEES Board of Directors, responsible for setting policies and strategy, making decisions, and governance. Several standing committees report to the Board of Directors. The Information Technology Strategy Committee reviews IT operations and performance. The Site Operations Committee reviews operations and performance of the engineering equipment sites. The Education, Outreach, and Training Committee reviews all outreach efforts across the NEES groups and stakeholders. To analyze the data and address our research questions, we chose network analysis. This approach allows us to see the formation of different subgroups in the communication and information exchanges among the stakeholders. The pattern of communication flows is treated as an indicator of the information and knowledge exchanged among those individuals. The denser the connections, the more ‘seamless’ the communication and flow of knowledge is assumed to be among individuals in the same subgroup. A symmetric matrix was constructed for each year (2003-04, 2005, 2006) using the data collected. Density scores were assigned based on the number of times individuals interacted with each other. For purposes of this research, an interaction is defined as either direct interaction via attendance at the same meeting or indirect interaction through reference in any way at a particular meeting. The density of the network itself was represented by the number of times an individual either directly or indirectly interacted with another individual at a meeting. Thus if individual A attended the same meeting with individual B on four separate occasions, the score would be set as four (4) in the network matrix. After generating the matrices and calculating the density scores, the data were imported into UCINET for network analysis (Scott 2002). Groupings of individuals were determined using the k-core method, which identifies subsets or groups within the network. K-cores are not necessarily methodologically cohesive subsets but they do identify areas of the graph which contain clique-like structures. A k-core is a maximal group of actors, all of whom are connected to at least k other members of the group (Hanneman and Riddle 2005). Connections that form distinct clusters indicate that communication among the stakeholders has split into subgroups based on one or more attributes. Individuals with a common in-group membership tend to favor and cooperate with members of their ingroup as opposed to those outside their group (Abrams et al 1990; Tajfel and Turner 1986). The presence of distinct clusters (in theory, subgroups; operationally, k-cores) indicates that more information is exchanged among those having similar expertise and/or similar affiliation. The character of the different clusters (subgroups, k-cores) indicates that a certain faultline (e.g., expertise, affiliation, gender) is ‘active’ in structuring the communication flows among the stakeholders (Lau and Murnighan, 1998). The faultlines serve to surface the attributes along which conflict (or lack of communication) has occurred and allow us to identify the type of diversity that has been associated with the conflict. To answer our research questions, and as depicted in Table 1, we coded each sub-group identified through the k-core analysis for each of the three time periods (2003-04, 2005, and 2006) along six attributes: primary expertise, affiliation, region, ethnicity, gender, and age. We reasoned that faultlines could occur across any of these attributes. For example, given that expertise is a task-relevant attribute and functions as a distinguishing characteristic of individuals involved in the coordination of the NEES project, communication flows among stakeholders could be affected by their expertise. In particular, the expertise attribute could function as a faultline, aligning the group of stakeholders into subgroups of similar expertise. Similarly, given that institutional affiliation is a task-relevant attribute (NEES is a collective project of many institutions) and that it functions as a distinguishing characteristic of individuals involved in the coordination of the NEES project, communication flows among stakeholders might be affected by their affiliation. In that case, the affiliation attribute would function as a faultline, aligning the group of stakeholders into subgroups of similar affiliation. Prior research suggests that gender, age, ethnicity, and geographic region may also act as faultlines in groups (e.g., Lau and Murnighan 2005, 1998); thus we also coded our groups along these attributes. Our coding rules are shown in Table 1. For each of the individuals in each year, centrality measures were also calculated and used to identify patterns within the data (Scott 2002). Individual measures were averaged to arrive at one score for each k-core group. As seen in Table 1, betweenness, which is the degree to which individuals are between others in the network and act as bridges to other areas of the network, was calculated. In addition, we use the eigenvector centrality measure which measures the importance of a node (individual) within the network. Approaches to Information Systems Development Track Twenty-eighth International Conference on Information Systems Montreal 2007 4 Results The network analysis yielded a set of distinct groups in each time period. As shown in Table 1, in the 2003-04 timeframe, there are three groups. In 2005, there are five k-core groups each with more than five members. In 2006, there are four k-core groups with more than five members. In each time period, the largest group (k-core id 25, 43, and 46) represents the core group. To answer our first research question, we examined the attributes of the groups in Table 1 to determine where the faultlines occur. Looking first at 2003-04, we observe that the three k-core groups have distinct characteristics. Group 12 appears to have strongly defined attributes, consisting of younger, white male academics, who are engineers, many of whom are from regions outside the West. In contrast, Group 25 (the core group) also is dominated by engineers and white males, but is older, includes fewer academics and more individuals from the West. Finally, Group 17 is more diverse in terms of primary expertise and academic affiliation and also has the highest proportion of females. In 2005, we observe a sub-group (Group 13) that is similar to Group 12 in 2003-4 in terms of having the same kinds of distinct faultlines (i.e., young, white male academics who are engineers from outside the West). In contrast, Group 43, the core group, appears more diverse than Group 25 (the core group in 2003-4), with a smaller proportion of engineers, more non-academics, even distribution across regions, more nonCaucasians and more females. This pattern continues in 2006, as the core group (Group 46) is similar to Group 43 (the core group in 2005), and both core groups are more diverse (fewer faultlines) than the core group in 2003-4 (Group 25). We also observe a subgroup in 2006 (Group 10) which has similar strongly defined attributes as Group 13 in 2005 and Group 12 in 2003-4 (i.e., younger, white male engineers who are academics). Generally, we find that faultlines are present for all of the dimensions we had identified (expertise, academic affiliation, region, ethnicity, gender and age). However, faultlines also appear to be more distinct for some subgroups than others. The second research question asks how the faultlines change as the project evolves. We consider this first for the core group. It appears as though the faultlines for expertise, affiliation, ethnicity, and gender grow less salient for the core group, as each of these becomes more diverse over time. There appears to be no difference in region over time. The higher standard deviation for age over time reinforces that the core group is becoming more diverse. Finally, the core group becomes progressively less “central” over time. As seen in Table 1, both centrality measures of the largest k-core group decreased over the years. Specifically, the normalized eigenvector value for 2003-04 is 24.48, for 2005 is 16.77, and for 2006 is 16.31. The normalized values for betweenness for 2003-04, 2005, and 2006 are 1.68, 1.19, and 1.07, respectively. This suggests that the largest k-core group became less central in the network over time. Thus, for the core group, faultlines seem to dissipate as the project unfolds from 2003-04 until 2006. It is also interesting to consider how faultlines remain or change for other groups. For example, group 12 in 2003-04, group 13 in 2005, and group 10 in 2006 consist almost entirely of Site Operation Committee members. Though the actual members changed over the years, some of the group characteristics are well-preserved: predominantly white male academics with expertise in earthquake engineering. This group significantly decreases in centrality over time, specifically, the normalized eigenvector value changes from 5.41 to 2.50 to 0.76 from 200304 to 2006 (see Table 1). Members of the Education, Outreach, and Training Committee also mostly belong to the same group in each year (group 12 in 2005 and group 16 in 2006). Neither group is central to the whole community. In general, the Board of Directors and IT Strategy Committee members are central to the whole community and have high degrees of connections with others. Also, the IT Strategy Committee becomes more central over time. In the largest group in 2003-2004, 7.69% (2 out of 26) are on the IT Strategy Committee. This ratio increased to 20.5% (9 out of 44) in 2005, and then to 29.8% (14 out of 47) in 2006. The Board of Directors still accounts for most of the largest group (32% in 2005 and 34% in 2006). To get a visual sense of the groups and faultlines, we used multidimensional scaling (MDS) and generated network graphs of the largest k-core groups over time. MDS provides visual representations of the proximities or similarities among the NEES stakeholders. We generated three network graphs of the k-core groups for the 2003-4 (Figure 1), 2005 (Figure 2) and 2006 periods (Figure 3). As can be observed in the graphs, the core group progressively becomes more and more inclusive, as the network becomes denser and includes more nodes (individuals) over time. Further, the individuals included in the core group in the later time periods differ in their attributes, i.e., they are less similar to each other in terms of their gender, expertise, affiliation, etc., than individuals in the core group in the earlier time period. This complements our observation that the core group is progressively becoming more diverse, and the faultlines less salient. 5 Groups with fewer than 5 members were suppressed for this analysis. Approaches to Information Systems Development Track Twenty-eighth International Conference on Information Systems Montreal 2007 5 Figure 1. 2003-4 Main K-Core Groups Figure 2. 2005 Main K-Core Groups Figure 3. 2006 Main K-Core Groups Discussion The purpose of this research was to examine the participation of a diverse body of stakeholders in a largescale, multi-party government-academic-industrial project over a four-year timeframe. The context of our study was the NEES project, a large, complex, distributed project funded by NSF and targeted for the earthquake engineering community. Stakeholders include academics from a variety of disciplines, government officials, and private sector participants. Using the theoretical lens of faultlines, we studied whether faultlines would emerge across a number of group attributes (expertise, affiliation, geographic region, ethnicity, gender, and age). We also examined how faultlines changed as the NEES project unfolded from 2003 until 2006. We found the presence of faultlines for various attributes like expertise, affiliation, region, and demographics. It is perhaps not surprising that some of these faultlines would occur. For example, affiliation and expertise diversity may both affect the group of stakeholders through social categorization. Stakeholders are affiliated with different institutions, each one with its distinct identity, norms, and values; thus, individuals may find it easier to work with others from the same affiliation. Similarly, stakeholders have different expertise; the existence of this faultline suggests that stakeholders align themselves into subgroups of similar expertise. The difficulty with the subgroups – indicated in our analysis – is that there is limited information exchange across subgroups and potentially ineffective utilization of team resources and knowledge. The results suggest that individuals with the same affiliation, or individuals with the same expertise, are communicating more often and sharing more information with individuals with a similar background. It is clear from our results that faultlines changed over the course of the project. In particular, we note that the faultlines for the core group become progressively less salient. Since the literature on faultlines suggests that Approaches to Information Systems Development Track Twenty-eighth International Conference on Information Systems Montreal 2007 6 strong faultlines are associated with group conflict, the dissipation of faultlines in the core group of the project we studied could be a positive signal that the diverse stakeholders are collaborating and working together. Given the nature of the project, broad participation from diverse stakeholders is especially critical to project success. Conclusions and Future Research In this study, we examined the formation and evolution of faultlines in a large-scale, multi-party ISD project that included the development and implementation of information systems and technologies along with earthquake engineering sites and technologies. As information systems development efforts become more complex, more uncertain, larger in scale and scope, as well as integrated even more into everyday products and services, they increasingly involve a diverse set of stakeholders who are often geographically distributed. The existing ISD research has often focused on projects with very different characteristics. Further, we know that ISD projects that are large and complex, that have uncertain or dynamic requirements and that span multiple, diverse stakeholder groups are especially challenging to manage (Brooks 1995). Thus, there is a critical need to understand how such difficult ISD projects can be successfully conducted. The research-in-progress reported here is a first step in an attempt to integrate the social network and social psychological perspectives in examining the process of large scale, multi-party information systems development. Our research has the potential to make a significant contribution to the ISD literature by revealing how diverse groups of stakeholders work together in very large, complex and distributed ISD projects. The next phases in our research include additional investigation of faultlines in ISD projects. Specifically, we plan to further explore the intriguing finding that project faultlines dissipated for the core group in the ISD project, and will examine what triggered the changes in the faultlines for the core group over the course of the project. Another interesting question is which faultlines are the most disruptive. We found the presence of faultlines for expertise, affiliation, region, gender, age, and ethnicity. However, it is possible that some faultlines have more impact than others on the conduct of the project. Finally, we plan to examine the impact of faultlines, and their evolution, on project success. Approaches to Information Systems Development Track Twenty-eighth International Conference on Information Systems Montreal 2007 7 Table 1: Group Characteristics by Year and K-Core Group (for K-Core Groups with More than 5 Nodes) Year K-Core ID (# in group) Primary Expertise* Affiliation** Region of the USA Ethnicity Gender Avg. Est. Age*** Avg. Norm. Eigenvector Avg. Norm. Betweenness 2003-04 25 (26) 77% EE-Related 8% Other 15% unknown 69% Academic 31% Non-Acad 54% West 46% Non-West 77% Caucasian 8% Other 15% unknown 85% M 15% F 53.16 SD=8.60 27% unknown 24.48 SD=6.25 1.68 SD=2.65 2003-04 17 (20) 60% EE-Related 20% Other 20% unknown 65% Academic 35% Non-Acad 40% West 60% Non-West 80% Caucasian 10% Other 10% unknown 60% M 40% F 52.50 SD=8.45 50% unknown 12.08 SD=3.49 0.61 SD=1.13 2003-04 12 (7) 100% EE-Related 100% Academic 29% West 71% Non-West 86% Caucasian